Current skill teaching methods of manipulator mainly construct a virtual space through three-dimensional reconstruction technology for manipulator to simulate and train. However， due to the different visual angles between human and manipulator， the traditional visual information reconstruction methods have large reconstruction errors， long time， and need harsh experimental environment and many sensors， so that the skills learned by manipulator in virtual space can not be well transferred to the real environment. To solve the above problems， a visual information real-time reconstruction method for manipulator operation was proposed. Firstly， information was extracted from real-time RGB images through Mask-Region Convolutional Neural Network（Mask-RCNN）. Then， the extracted RGB images and other visual information were jointly encoded， and the visual information was mapped to the three-dimensional position information of the manipulator operation space through Residual Neural Network-18 （ResNet-18）. Finally， an outlier adjustment method based on Cluster Center DIStance constrained （CC-DIS） was proposed to reduce the reconstruction error， and the adjusted position information was visualized by Open Graphics Library （OpenGL）. In this way， the three-dimensional real-time reconstruction of the manipulator operation space was completed. Experimental results show that the proposed method has high reconstruction speed and reconstruction accuracy. It only takes 62.92 milliseconds to complete a three-dimensional reconstruction with a reconstruction speed of up to 16 frames per second and a reconstruction relative error of about 5.23%. Therefore， it can be effectively applied to the manipulator skill teaching tasks.